Repository navigation
Improve missing-metric incident-onset evidence - #104
Open
JeffPengCoder wants to merge 3 commits into
Open
JeffPengCoder wants to merge 3 commits into
JeffPengCoder wants to merge 3 commits into
Conversation
JeffPengCoder
marked this pull request as ready for review
November 9, 2026 21:59
JeffPengCoder
requested review from
RohanAdwankar and
kgengedullay
as code owners
November 9, 2026 21:59
Refs NVIDIA#103. Preserve range boundaries and source timestamps, add a bounded read-only last-seen investigation, and wire its skill, permissions, Helm assets, and synthetic regression coverage. Signed-off-by: JeffPengCoder <jepeng@nvidia.com>
Signed-off-by: JeffPengCoder <jepeng@nvidia.com>
Signed-off-by: JeffPengCoder <jepeng@nvidia.com>
JeffPengCoder
force-pushed
the
fix/observability-incident-onset
branch
from
November 19, 2026 06:10
415edc2 to
4e6c153
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Refs #103
Draft: make missing-metric onset investigation use source-sample evidence, not
the boundaries of the last chart query. This is not a claim that every model
or the complete native OpenClaw workflow now respects the skill.
An empty recent range does not establish when data stopped. There are also two
lower-level pitfalls: the current stride compaction can omit the final returned
point, and
query_rangeevaluation times can continue past the last originalsample because of Prometheus lookback. Simply widening a chart query does not
resolve those ambiguities.
This change keeps one workflow together: historical evidence for a missing
metric, its read-only investigation tool, the skill instructions consuming that
evidence, and the runtime wiring and tests required to ship it.
downsampling and returned boundary timestamps.
Add explicit ISO formatting for timestamp-valued queries without guessing that
ordinary metric values are timestamps.
observability_metric_last_seen. It searches adjacent raw range vectorsthrough the instant query API, retaining the metric and exact source labels.
It returns the last original sample, actual checked windows, elapsed time, and
an optional cadence-qualified first-missing-expected-sample estimate.
2 MiB per response. Default history is seven days; maximum requested history
is 30 days. Ambiguous series, errors, partial responses and unsupported result
types do not become evidence of missing data.
distinguish sample age from searched coverage, and keep entirely empty
history unknown. No prior existence or retention claim is inferred.
contract, honor tool enable/disable controls, and classify it as read-only for
View/automation policy. No new infrastructure mutation permission is added.
Validation
Local runtime: Node.js 25.8.1, Helm 3.19.0, Prometheus 3.16.0. Chart dependencies
use the public OpenShell 0.0.75 chart and the same minimal IRAOP stub as CI.
The two skips are native OpenClaw memory integration and a real VictoriaMetrics
endpoint. The real Prometheus history test was enabled and covered all four
synthetic cases. The fixture generator emits all 30121 original samples, with
no scrape targets or stale markers. Setup commands and every fixture row are in
helm/mosaic-stack/tests/metric-history.md.Deterministic tests also cover adjacent
(start,end]windows, a sample on awindow boundary, zero and non-finite values, unknown cadence, empty history,
request/time/size bounds, multiple series, label escaping, invalid input,
transport failures, partial warnings, malformed samples, module rendering and
read-only access controls.
Repeatable model/harness backtest
Add a portable runner and reproduction guide for comparing two source revisions with the same served model. It covers all four fixture cases plus the empty-window onset follow-up, with repeated runs and full request, response, tool, backend and answer records.
The runner validates all 30,121 fixture samples, records source/model/server identities and inference settings, and rejects unmatched comparisons. Normal inference-cache behavior is the default; cache isolation is a separately labeled diagnostic. Completion and tool invocation are execution observations, not factual-accuracy grades; every answer remains subject to the documented case-specific review.
Seven new runner tests pass. Both the PR base and candidate were additionally exercised against real Prometheus with an explicitly synthetic model stub to verify the replay plumbing. That smoke test does not measure a real model. The requested real Nemotron comparison has now completed, as recorded below. The older Qwen results remain separate evidence; native OpenClaw acceptance is still outstanding.
Nemotron 3 Super NVFP4: observed backtest
The complete run record and all final answers, token usage, verdicts and provenance are checked in. No case rows or failed attempts are silently removed.
nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4, revisionff433f5493e25d631c9f12b5d55c674229923d02; this is 3 Super, not 3.5 Super VL.qwen3_coderand checkpoint-pinnedsuper_v3parsers.307afa7e485a334389b4f2c8c68128fbf84ff007, candidate8305d59e44470e10d268a4e60fc36870929b2e27. The subsequent result-record commit changes documentation only.Codex reviewed the full answers against the pre-run rubric; this is not a blinded human evaluation or a general model accuracy score. The complete per-case table, exact durations, all per-turn token records and both score definitions are in the linked record. The suite uses a binary rubric, so its raw 0/1 mean equals its binary full-success rate.
The candidate improves source-sample retrieval, but it still sometimes stops a follow-up at an empty 48-hour window, omits the supplied cadence, conflates a last-present sample with the first expected absence, or adds unsupported workload/collector claims. One response recovered the right dates but contradicted itself with “three hours” versus “three days.” The three-day-gap seed-2 pass carries an explicit precision note in the record. These results support keeping this PR Draft and #103 open until native-runtime investigation completion and evidence-checked final output are verified. Thinking-enabled behavior was not measured.
Behavior evidence and draft limitations
An isolated model/tool replay used
nvidia/Qwen3.6-35B-A3B-NVFP4, a fixedinvestigation time of
2026-11-06T00:00:00Z, and the real synthetic Prometheusfixture. The skill was explicitly loaded; this does not test native skill
selection, complete OpenClaw prompting, or the Mosaic UI.
With the same request-isolated inference control on both versions, the upstream
follow-up case incorrectly dated a three-day gap to the start of the recent
two-hour query,
2026-11-05T22:00:00Z. The candidate's two follow-up repetitionsboth invoked the new tool, returned the actual last sample
2026-11-02T23:59:00Z, and qualified2026-11-03T00:00:00Zas the first expectedmissing sample at the known 60-second cadence, not a physical fault timestamp.
All seven final candidate replays completed after one investigation tool call
and two model turns, but not every prose claim passed review:
Earlier shared-cache replays also showed unstable tool use. An identical-input
six-request probe produced tool calls with fresh cache salts and on a warm
same-salt isolated prefix, while the two shared-prefix probes did not call a
tool. That is an inference-environment confound, not proof of the root cause of
#103. Both sides of the controlled comparison used a fresh per-request
cache_salt. No inference-server configuration or cache workaround is changedin this PR, and raw model logs and deployment configuration are not included.
The new tool requires raw Prometheus range-vector semantics. It fails closed
for evaluated-vector substitutes and native histograms; it does not locate
threshold breaches or previous recovered gaps. Generic query tools remain
available for those investigations. Maintainer feedback on the dedicated
read-only tool/API is requested before treating this draft as merge-ready.
What remains before this can close the issue
This patch repairs the evidence-producing layer and adds a bounded investigation
path. It does not enforce the agent's decision to use that path or validate
every claim in its final natural-language answer. Passing tool tests and CI is
therefore not the same as fully addressing the reported behavior. Further work is feasible,
but should be verified in the actual conversation runtime:
selection and the complete Mosaic/OpenClaw conversation, particularly a
follow-up after an empty recent query. The response must either be grounded
in same-source historical evidence or identify an actual search/access limit.
Merely suggesting a still-available historical query is not completion.
elapsed age, checked coverage and any cadence-qualified missing-time estimate
from structured tool evidence. Do not let the model substitute a query
boundary, recompute a contradictory duration, claim retention from an empty
result, or turn metric presence into proof of collector/workload health.
This runtime/output enforcement is not implemented by the current patch.
follow-up, no available history, a fresh zero, a shorter gap, ambiguous
sources and backend errors. Include the normal inference configuration, not
only request-cache-isolated replay, and record remaining failures.
The observed inference-cache sensitivity should be investigated separately. It
is not evidence that the issue reporter had the same serving problem, and no
serving configuration change should be implied by merging this patch.
For now this PR references the issue without a closing keyword. Keep #103 open
until the native-runtime and factual-output acceptance criteria are satisfied;
the reference can then be converted to a closing link if maintainers agree.
Checklist